{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intepretability on Hateful Twitter Datasets\n",
    "\n",
    "In this demo, we apply saliency maps (with support of sparse tensors) on the task on the detection of Twitter users who use hateful lexicon using graph machine learning with Stellargraph.\n",
    "\n",
    "We consider the use-case of identifying hateful users on Twitter motivated by the work in [1] and using the dataset also published in [1]. Classification is based on a graph based on users' retweets and attributes as related to their account activity, and the content of tweets.\n",
    "\n",
    "We pose identifying hateful users as a binary classification problem. We demonstrate the advantage of connected vs unconnected data in a semi-supervised setting with few training examples.\n",
    "\n",
    "For connected data, we use Graph Convolutional Networks [2] as implemented in the `stellargraph` library. We pose the problem of identifying hateful tweeter users as node attribute inference in graphs.\n",
    "\n",
    "We then use the interpretability tool (i.e., saliency maps) implemented in our library to demonstrate how to obtain the importance of the node features and links to gain insights into the model.\n",
    "\n",
    "**References**\n",
    "\n",
    "1. \"Like Sheep Among Wolves\": Characterizing Hateful Users on Twitter. M. H. Ribeiro, P. H. Calais, Y. A. Santos, V. A. F. Almeida, and W. Meira Jr.  arXiv preprint arXiv:1801.00317 (2017).\n",
    "\n",
    "\n",
    "2. Semi-Supervised Classification with Graph Convolutional Networks. T. Kipf, M. Welling. ICLR 2017. arXiv:1609.02907 \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "import itertools\n",
    "import os\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.linear_model import LogisticRegressionCV\n",
    "\n",
    "import stellargraph as sg\n",
    "from stellargraph.mapper import GraphSAGENodeGenerator, FullBatchNodeGenerator\n",
    "from stellargraph.layer import GraphSAGE, GCN, GAT\n",
    "from stellargraph import globalvar\n",
    "\n",
    "from tensorflow.keras import layers, optimizers, losses, metrics, Model, models\n",
    "from sklearn import preprocessing, feature_extraction\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from scipy.sparse import csr_matrix, lil_matrix\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "def remove_prefix(text, prefix):\n",
    "    return text[text.startswith(prefix) and len(prefix):]\n",
    "\n",
    "def plot_history(history):\n",
    "    metrics = sorted(set([remove_prefix(m, \"val_\") for m in list(history.history.keys())]))\n",
    "    for m in metrics:\n",
    "        # summarize history for metric m\n",
    "        plt.plot(history.history[m])\n",
    "        plt.plot(history.history['val_' + m])\n",
    "        plt.title(m, fontsize=18)\n",
    "        plt.ylabel(m, fontsize=18)\n",
    "        plt.xlabel('epoch', fontsize=18)\n",
    "        plt.legend(['train', 'validation'], loc='best')\n",
    "        plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train GCN model on the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = os.path.expanduser(\"~/data/hateful-twitter-users\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First load and prepare the node features\n",
    "\n",
    "Each node in the graph is associated with a large number of features (also referred to as attributes). \n",
    "\n",
    "The list of features is given [here](https://www.kaggle.com/manoelribeiro/hateful-users-on-twitter). We repeated here for convenience.\n",
    "\n",
    "hate :(\"hateful\"|\"normal\"|\"other\")\n",
    "  if user was annotated as hateful, normal, or not annotated.\n",
    "  \n",
    "  (is_50|is_50_2) :bool\n",
    "  whether user was deleted up to 12/12/17 or 14/01/18. \n",
    "  \n",
    "  (is_63|is_63_2) :bool\n",
    "  whether user was suspended up to 12/12/17 or 14/01/18. \n",
    "        \n",
    "  (hate|normal)_neigh :bool\n",
    "  is the user on the neighborhood of a (hateful|normal) user? \n",
    "  \n",
    "  [c_] (statuses|follower|followees|favorites)_count :int\n",
    "  number of (tweets|follower|followees|favorites) a user has.\n",
    "  \n",
    "  [c_] listed_count:int\n",
    "  number of lists a user is in.\n",
    "\n",
    "  [c_] (betweenness|eigenvector|in_degree|outdegree) :float\n",
    "  centrality measurements for each user in the retweet graph.\n",
    "  \n",
    "  [c_] *_empath :float\n",
    "  occurrences of empath categories in the users latest 200 tweets.\n",
    "\n",
    "  [c_] *_glove :float          \n",
    "  glove vector calculated for users latest 200 tweets.\n",
    "  \n",
    "  [c_] (sentiment|subjectivity) :float\n",
    "  average sentiment and subjectivity of users tweets.\n",
    "  \n",
    "  [c_] (time_diff|time_diff_median) :float\n",
    "  average and median time difference between tweets.\n",
    "  \n",
    "  [c_] (tweet|retweet|quote) number :float\n",
    "  percentage of direct tweets, retweets and quotes of an user.\n",
    "  \n",
    "  [c_] (number urls|number hashtags|baddies|mentions) :float\n",
    "  number of bad words|mentions|urls|hashtags per tweet in average.\n",
    "  \n",
    "  [c_] status length :float\n",
    "  average status length.\n",
    "  \n",
    "  hashtags :string\n",
    "  all hashtags employed by the user separated by spaces.\n",
    "  \n",
    "**Notice** that c_ are attributes calculated for the 1-neighborhood of a user in the retweet network (averaged out)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we are going to load the user features and prepare them for machine learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_feat = pd.read_csv(os.path.join(data_dir, \n",
    "                                      'users_neighborhood_anon.csv'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data cleaning and preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset as given includes a large number of graph related features that are manually extracted. \n",
    "\n",
    "Since we are going to employ modern graph neural networks methods for classification, we are going to drop these manually engineered features. \n",
    "\n",
    "The power of Graph Neural Networks stems from their ability to learn useful graph-related features eliminating the need for manual feature engineering."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_cleaning(feat):\n",
    "    feat = feat.drop(columns=[\"hate_neigh\", \"normal_neigh\"])\n",
    "    \n",
    "    # Convert target values in hate column from strings to integers (0,1,2)\n",
    "    feat['hate'] = np.where(feat['hate']=='hateful', 1, np.where(feat['hate']=='normal', 0, 2))\n",
    "    \n",
    "    # missing information\n",
    "    number_of_missing = feat.isnull().sum()\n",
    "    number_of_missing[number_of_missing!=0]\n",
    "    \n",
    "    # Replace NA with 0\n",
    "    feat.fillna(0, inplace=True)\n",
    "\n",
    "    # droping info about suspension and deletion as it is should not be use din the predictive model\n",
    "    feat.drop(feat.columns[feat.columns.str.contains(\"is_|_glove|c_|sentiment\")], axis=1, inplace=True)\n",
    "\n",
    "    # drop hashtag feature\n",
    "    feat.drop(['hashtags'], axis=1, inplace=True)\n",
    "\n",
    "    # Drop centrality based measures\n",
    "    feat.drop(columns=['betweenness', 'eigenvector', 'in_degree', 'out_degree'], inplace=True)\n",
    "    \n",
    "    feat.drop(columns=['created_at'], inplace=True)\n",
    "    \n",
    "    return feat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "node_data = data_cleaning(users_feat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The continous features in our dataset have distributions with very long tails. We apply normalization to correct for this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ignore the first two columns because those are user_id and hate (the target variable)\n",
    "df_values = node_data.iloc[:, 2:].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "pt = preprocessing.PowerTransformer(method='yeo-johnson', \n",
    "                                    standardize=True) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_values_log = pt.fit_transform(df_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "node_data.iloc[:, 2:] = df_values_log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set the dataframe index to be the same as the user_id and drop the user_id columns\n",
    "node_data.index = node_data.index.map(str)\n",
    "node_data.drop(columns=['user_id'], inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next load the graph\n",
    "\n",
    "Now that we have the node features prepared for machine learning, let us load the retweet graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_nx = nx.read_edgelist(path=os.path.expanduser(os.path.join(data_dir,\n",
    "                                                             \"users.edges\")))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100386, 2194979)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "g_nx.number_of_nodes(), g_nx.number_of_edges()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph has just over 100k nodes and approximately 2.2m edges.\n",
    "\n",
    "We aim to train a graph neural network model that will predict the \"hate\"attribute on the nodes.\n",
    "\n",
    "For computation convenience, we have mapped the target labels **normal**, **hateful**, and **other** to the numeric values **0**, **1**, and **2** respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0, 1, 2}\n"
     ]
    }
   ],
   "source": [
    "print(set(node_data[\"hate\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['10999', '55317', '44622', '71821', '57907', '4806', '95121', '36794',\n",
       "       '19152', '253',\n",
       "       ...\n",
       "       '23778', '32487', '41151', '97765', '45269', '37714', '1790', '54923',\n",
       "       '38512', '81763'],\n",
       "      dtype='object', length=100386)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "node_data = node_data.loc[list(g_nx.nodes())]\n",
    "node_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For machine learning we want to take a subset of the nodes for training, and use the rest for validation and testing. We'll use scikit-learn again to split our data into training and test sets.\n",
    "\n",
    "The total number of annotated nodes is very small when compared to the total number of nodes in the graph. We are only going to use 15% of the annotated nodes for training and the remaining 85% of nodes for testing.\n",
    "\n",
    "First, we are going to select the subset of nodes that are annotated as hateful or normal. These will be the nodes that have 'hate' values that are either 0 or 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# choose the nodes annotated with normal or hateful classes\n",
    "annotated_users = node_data[node_data['hate']!=2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "annotated_user_features = annotated_users.drop(columns=['hate'])\n",
    "annotated_user_targets = annotated_users[['hate']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 4971 annoted nodes out of a possible, approximately, 100k nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    4427\n",
      "1     544\n",
      "Name: hate, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(annotated_user_targets.hate.value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sizes and class distributions for train/test data\n",
      "Shape train_data (745, 204)\n",
      "Shape test_data (4226, 204)\n",
      "Train data number of 0s 667 and 1s 78\n",
      "Test data number of 0s 3760 and 1s 466\n"
     ]
    }
   ],
   "source": [
    "# split the data\n",
    "train_data, test_data, train_targets, test_targets = train_test_split(annotated_user_features,\n",
    "                                         annotated_user_targets,\n",
    "                                         test_size=0.85,\n",
    "                                         random_state=101)\n",
    "train_targets = train_targets.values\n",
    "test_targets = test_targets.values\n",
    "print(\"Sizes and class distributions for train/test data\")\n",
    "print(\"Shape train_data {}\".format(train_data.shape))\n",
    "print(\"Shape test_data {}\".format(test_data.shape))\n",
    "print(\"Train data number of 0s {} and 1s {}\".format(np.sum(train_targets==0), \n",
    "                                                    np.sum(train_targets==1)))\n",
    "print(\"Test data number of 0s {} and 1s {}\".format(np.sum(test_targets==0), \n",
    "                                                   np.sum(test_targets==1)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((745, 1), (4226, 1))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_targets.shape, test_targets.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((745, 204), (4226, 204))"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data.shape, test_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to use 745 nodes for training and 4226 nodes for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# choosing features to assign to a graph, excluding target variable\n",
    "node_features = node_data.drop(columns=['hate'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dealing with imbalanced data\n",
    "\n",
    "Because the training data exhibit high imbalance, we introduce class weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{0: 0.5584707646176912, 1: 4.7756410256410255}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.utils.class_weight import compute_class_weight\n",
    "class_weights = compute_class_weight('balanced', \n",
    "                                     np.unique(train_targets), \n",
    "                                     train_targets[:,0])\n",
    "train_class_weights = dict(zip(np.unique(train_targets), \n",
    "                               class_weights))\n",
    "train_class_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our data is now ready for machine learning.\n",
    "\n",
    "Node features are stored in the Pandas DataFrame `node_features`.\n",
    "\n",
    "The graph in networkx format is stored in the variable `g_nx`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify global parameters\n",
    "\n",
    "Here we specify some parameters that control the type of model we are going to use. For example, we specify the base model type, e.g., GCN, GraphSAGE, etc, as well as model-specific parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 20  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the base graph machine learning model in Keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a `StellarGraph` object from the `NetworkX` graph and the node features and targets. It is `StellarGraph` objects that we use in this library to perform machine learning tasks on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['10999', '55317', '44622', '71821', '57907', '4806', '95121', '36794', '19152', '253']\n"
     ]
    }
   ],
   "source": [
    "G = sg.StellarGraph(g_nx, node_features=node_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To feed data from the graph to the Keras model we need a generator. The generators are specialized to the model and the learning task. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For training we map only the training nodes returned from our splitter and the target values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using GCN (local pooling) filters...\n"
     ]
    }
   ],
   "source": [
    "generator = FullBatchNodeGenerator(G, method=\"gcn\", sparse=True)\n",
    "train_gen = generator.flow(train_data.index, train_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "base_model = GCN(\n",
    "    layer_sizes=[32, 16],\n",
    "    generator = generator,\n",
    "    bias=True,\n",
    "    dropout=0.5,\n",
    "    activations=[\"elu\", \"elu\"]\n",
    ")\n",
    "x_inp, x_out = base_model.build()\n",
    "prediction = layers.Dense(units=1, activation=\"sigmoid\")(x_out)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create a Keras model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's create the actual Keras model with the graph inputs `x_inp` provided by the `base_model` and outputs being the predictions from the softmax layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Model(inputs=x_inp, outputs=prediction)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We compile our Keras model to use the `Adam` optimiser and the binary cross entropy loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(\n",
    "    optimizer=optimizers.Adam(lr=0.005),\n",
    "    loss=losses.binary_crossentropy,\n",
    "    metrics=[\"acc\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<keras.engine.training.Model at 0x7f2aa36a4208>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the model, keeping track of its loss and accuracy on the training set, and its performance on the test set during the training. We don't use the test set during training but only for measuring the trained model's generalization performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_gen = generator.flow(test_data.index, test_targets)\n",
    "history = model.fit_generator(\n",
    "    train_gen,\n",
    "    epochs=epochs,\n",
    "    validation_data=test_gen,\n",
    "    verbose=2,\n",
    "    shuffle=False,\n",
    "    class_weight=None,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have trained the model, let's evaluate it on the test set.\n",
    "\n",
    "We are going to consider 4 evaluation metrics calculated on the test set: Accuracy, Area Under the ROC curve (AU-ROC), the ROC curve, and the confusion table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Test Set Metrics:\n",
      "\tloss: 0.2787\n",
      "\tacc: 0.9188\n"
     ]
    }
   ],
   "source": [
    "test_metrics = model.evaluate_generator(test_gen)\n",
    "print(\"\\nTest Set Metrics:\")\n",
    "for name, val in zip(model.metrics_names, test_metrics):\n",
    "    print(\"\\t{}: {:0.4f}\".format(name, val))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_nodes = node_data.index\n",
    "all_gen = generator.flow(all_nodes)\n",
    "all_predictions = model.predict_generator(all_gen).squeeze()[..., np.newaxis]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100386, 1)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_predictions.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_predictions_df = pd.DataFrame(all_predictions, \n",
    "                                  index=node_data.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's extract the predictions for the test data only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = all_predictions_df.loc[test_data.index, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4226, 1)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_preds.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predictions are the probability of the true class that in this case is the probability of a user being hateful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The AUC on test set:\n",
      "\n",
      "0.8723021870148846\n"
     ]
    }
   ],
   "source": [
    "test_predictions = test_preds.values\n",
    "test_predictions_class = ((test_predictions>0.5)*1).flatten()\n",
    "test_df = pd.DataFrame({\"Predicted_score\": test_predictions.flatten(), \n",
    "                        \"Predicted_class\": test_predictions_class, \n",
    "                        \"True\": test_targets[:,0]})\n",
    "roc_auc = metrics.roc_auc_score(test_df['True'].values, \n",
    "                                test_df['Predicted_score'].values)\n",
    "print(\"The AUC on test set:\\n\")\n",
    "print(roc_auc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Interpretability by Saliency Maps\n",
    "\n",
    "To understand which features and edges the model is looking at while making the predictions, we use the interpretability tool in the StellarGraph library (i.e., saliency maps) to demonstrate the importance of node features and edges given a target user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from stellargraph.utils.saliency_maps import IntegratedGradients\n",
    "int_saliency = IntegratedGradients(model, all_gen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[34474, 52844, 36367, 71889, 12850, 54770, 61878, 35677]\n",
      "target_idx = 36367, target_nid = 77692\n",
      "prediction score for node 36367 is [0.9631094]\n",
      "ground truth score for node 36367 is [1]\n"
     ]
    }
   ],
   "source": [
    "from stellargraph.utils.saliency_maps import IntegratedGradients, GradientSaliency\n",
    "#we first select a list of nodes which are confidently classified as hateful.\n",
    "predicted_hateful_index = set(np.where(all_predictions > 0.9)[0].tolist())\n",
    "test_indices_set = set([int(k) for k in test_data.index.tolist()])\n",
    "hateful_in_test = list(predicted_hateful_index.intersection(test_indices_set))\n",
    "\n",
    "#let's pick one node from the predicted hateful users as an example.\n",
    "idx = 2\n",
    "target_idx = hateful_in_test[idx]\n",
    "target_nid = list(G.nodes())[target_idx]\n",
    "print('target_idx = {}, target_nid = {}'.format(target_idx, target_nid))\n",
    "print('prediction score for node {} is {}'.format(target_idx, all_predictions[target_idx]))\n",
    "print('ground truth score for node {} is {}'.format(target_idx, test_targets[test_data.index.tolist().index(str(target_nid))]))\n",
    "[X,all_targets,A_index, A], y_true_all = all_gen[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the prediction of the target node, we then calculate the importance of the features for each node in the graph. Our support for sparse saliency maps makes it efficient to fit the scale like this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We set the target_idx which is our target node. \n",
    "node_feature_importance = int_saliency.get_integrated_node_masks(target_idx, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As `node_feature_importance` is a matrix where `node_feature_importance[i][j]` indicates the importance of the j-th feature of node i to the prediction of the target node, we sum up the feature importance of each node to measure its node importance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.38610699  0.17760033  0.15617194 ... -0.03730714 -0.03870907\n",
      " -0.04825595]\n",
      "node_importance has 12721 non-zero values\n"
     ]
    }
   ],
   "source": [
    "node_importance = np.sum(node_feature_importance, axis=-1)\n",
    "node_importance_rank = np.argsort(node_importance)[::-1]\n",
    "print(node_importance[node_importance_rank])\n",
    "print('node_importance has {} non-zero values'.format(np.where(node_importance != 0)[0].shape[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We expect the number of non-zero values of `node_importance` to match the number of nodes in the ego graph. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The ego graph of the target node has 12721 neighbors\n"
     ]
    }
   ],
   "source": [
    "G_ego = nx.ego_graph(g_nx,target_nid, radius=2)\n",
    "print('The ego graph of the target node has {} neighbors'.format(len(G_ego.nodes())))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then analyze the feature importance of the top-250 important nodes. See the output for the top-5 importance nodes. For each row, the features are sorted according to their importance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>195</th>\n",
       "      <th>196</th>\n",
       "      <th>197</th>\n",
       "      <th>198</th>\n",
       "      <th>199</th>\n",
       "      <th>200</th>\n",
       "      <th>201</th>\n",
       "      <th>202</th>\n",
       "      <th>203</th>\n",
       "      <th>204</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>36367</td>\n",
       "      <td>morning_empath</td>\n",
       "      <td>wedding_empath</td>\n",
       "      <td>fear_empath</td>\n",
       "      <td>computer_empath</td>\n",
       "      <td>furniture_empath</td>\n",
       "      <td>legend_empath</td>\n",
       "      <td>cold_empath</td>\n",
       "      <td>time_diff_median</td>\n",
       "      <td>sleep_empath</td>\n",
       "      <td>...</td>\n",
       "      <td>phone_empath</td>\n",
       "      <td>listen_empath</td>\n",
       "      <td>tool_empath</td>\n",
       "      <td>lust_empath</td>\n",
       "      <td>programming_empath</td>\n",
       "      <td>hygiene_empath</td>\n",
       "      <td>hearing_empath</td>\n",
       "      <td>tweet number</td>\n",
       "      <td>retweet number</td>\n",
       "      <td>medieval_empath</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>34534</td>\n",
       "      <td>morning_empath</td>\n",
       "      <td>pet_empath</td>\n",
       "      <td>computer_empath</td>\n",
       "      <td>fear_empath</td>\n",
       "      <td>rage_empath</td>\n",
       "      <td>favorites_count</td>\n",
       "      <td>ridicule_empath</td>\n",
       "      <td>wedding_empath</td>\n",
       "      <td>tool_empath</td>\n",
       "      <td>...</td>\n",
       "      <td>weakness_empath</td>\n",
       "      <td>sound_empath</td>\n",
       "      <td>masculine_empath</td>\n",
       "      <td>statuses_count</td>\n",
       "      <td>positive_emotion_empath</td>\n",
       "      <td>negative_emotion_empath</td>\n",
       "      <td>followers_count</td>\n",
       "      <td>tweet number</td>\n",
       "      <td>retweet number</td>\n",
       "      <td>home_empath</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>69814</td>\n",
       "      <td>morning_empath</td>\n",
       "      <td>joy_empath</td>\n",
       "      <td>ridicule_empath</td>\n",
       "      <td>money_empath</td>\n",
       "      <td>rage_empath</td>\n",
       "      <td>programming_empath</td>\n",
       "      <td>lust_empath</td>\n",
       "      <td>anonymity_empath</td>\n",
       "      <td>science_empath</td>\n",
       "      <td>...</td>\n",
       "      <td>medieval_empath</td>\n",
       "      <td>fear_empath</td>\n",
       "      <td>government_empath</td>\n",
       "      <td>children_empath</td>\n",
       "      <td>computer_empath</td>\n",
       "      <td>youth_empath</td>\n",
       "      <td>musical_empath</td>\n",
       "      <td>body_empath</td>\n",
       "      <td>white_collar_job_empath</td>\n",
       "      <td>furniture_empath</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21749</td>\n",
       "      <td>pet_empath</td>\n",
       "      <td>rage_empath</td>\n",
       "      <td>legend_empath</td>\n",
       "      <td>wedding_empath</td>\n",
       "      <td>ridicule_empath</td>\n",
       "      <td>science_empath</td>\n",
       "      <td>furniture_empath</td>\n",
       "      <td>white_collar_job_empath</td>\n",
       "      <td>dispute_empath</td>\n",
       "      <td>...</td>\n",
       "      <td>leader_empath</td>\n",
       "      <td>tweet number</td>\n",
       "      <td>religion_empath</td>\n",
       "      <td>real_estate_empath</td>\n",
       "      <td>retweet number</td>\n",
       "      <td>strength_empath</td>\n",
       "      <td>zest_empath</td>\n",
       "      <td>sexual_empath</td>\n",
       "      <td>computer_empath</td>\n",
       "      <td>water_empath</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>69254</td>\n",
       "      <td>furniture_empath</td>\n",
       "      <td>morning_empath</td>\n",
       "      <td>ridicule_empath</td>\n",
       "      <td>legend_empath</td>\n",
       "      <td>science_empath</td>\n",
       "      <td>programming_empath</td>\n",
       "      <td>farming_empath</td>\n",
       "      <td>traveling_empath</td>\n",
       "      <td>writing_empath</td>\n",
       "      <td>...</td>\n",
       "      <td>weakness_empath</td>\n",
       "      <td>attractive_empath</td>\n",
       "      <td>valuable_empath</td>\n",
       "      <td>noise_empath</td>\n",
       "      <td>body_empath</td>\n",
       "      <td>ugliness_empath</td>\n",
       "      <td>surprise_empath</td>\n",
       "      <td>philosophy_empath</td>\n",
       "      <td>sadness_empath</td>\n",
       "      <td>pet_empath</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 205 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     0                 1               2                3                4    \\\n",
       "0  36367    morning_empath  wedding_empath      fear_empath  computer_empath   \n",
       "1  34534    morning_empath      pet_empath  computer_empath      fear_empath   \n",
       "2  69814    morning_empath      joy_empath  ridicule_empath     money_empath   \n",
       "3  21749        pet_empath     rage_empath    legend_empath   wedding_empath   \n",
       "4  69254  furniture_empath  morning_empath  ridicule_empath    legend_empath   \n",
       "\n",
       "                5                   6                 7    \\\n",
       "0  furniture_empath       legend_empath       cold_empath   \n",
       "1       rage_empath     favorites_count   ridicule_empath   \n",
       "2       rage_empath  programming_empath       lust_empath   \n",
       "3   ridicule_empath      science_empath  furniture_empath   \n",
       "4    science_empath  programming_empath    farming_empath   \n",
       "\n",
       "                       8               9    ...              195  \\\n",
       "0         time_diff_median    sleep_empath  ...     phone_empath   \n",
       "1           wedding_empath     tool_empath  ...  weakness_empath   \n",
       "2         anonymity_empath  science_empath  ...  medieval_empath   \n",
       "3  white_collar_job_empath  dispute_empath  ...    leader_empath   \n",
       "4         traveling_empath  writing_empath  ...  weakness_empath   \n",
       "\n",
       "                 196                197                 198  \\\n",
       "0      listen_empath        tool_empath         lust_empath   \n",
       "1       sound_empath   masculine_empath      statuses_count   \n",
       "2        fear_empath  government_empath     children_empath   \n",
       "3       tweet number    religion_empath  real_estate_empath   \n",
       "4  attractive_empath    valuable_empath        noise_empath   \n",
       "\n",
       "                       199                      200              201  \\\n",
       "0       programming_empath           hygiene_empath   hearing_empath   \n",
       "1  positive_emotion_empath  negative_emotion_empath  followers_count   \n",
       "2          computer_empath             youth_empath   musical_empath   \n",
       "3           retweet number          strength_empath      zest_empath   \n",
       "4              body_empath          ugliness_empath  surprise_empath   \n",
       "\n",
       "                 202                      203               204  \n",
       "0       tweet number           retweet number   medieval_empath  \n",
       "1       tweet number           retweet number       home_empath  \n",
       "2        body_empath  white_collar_job_empath  furniture_empath  \n",
       "3      sexual_empath          computer_empath      water_empath  \n",
       "4  philosophy_empath           sadness_empath        pet_empath  \n",
       "\n",
       "[5 rows x 205 columns]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names = annotated_users.keys()[1:].values\n",
    "feature_importance_rank = np.argsort(node_feature_importance[target_idx])[::-1]\n",
    "df = pd.DataFrame([([k] + list(feature_names[np.argsort(node_feature_importance[k])[::-1]])) for k in node_importance_rank[:250]], columns = range(205)) \n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check, we expect the target node itself to have a relatively high importance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.38610699226502143\n",
      "The node itself is the 1-th important node\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>194</th>\n",
       "      <th>195</th>\n",
       "      <th>196</th>\n",
       "      <th>197</th>\n",
       "      <th>198</th>\n",
       "      <th>199</th>\n",
       "      <th>200</th>\n",
       "      <th>201</th>\n",
       "      <th>202</th>\n",
       "      <th>203</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>morning_empath</td>\n",
       "      <td>wedding_empath</td>\n",
       "      <td>fear_empath</td>\n",
       "      <td>computer_empath</td>\n",
       "      <td>furniture_empath</td>\n",
       "      <td>legend_empath</td>\n",
       "      <td>cold_empath</td>\n",
       "      <td>time_diff_median</td>\n",
       "      <td>sleep_empath</td>\n",
       "      <td>vacation_empath</td>\n",
       "      <td>...</td>\n",
       "      <td>phone_empath</td>\n",
       "      <td>listen_empath</td>\n",
       "      <td>tool_empath</td>\n",
       "      <td>lust_empath</td>\n",
       "      <td>programming_empath</td>\n",
       "      <td>hygiene_empath</td>\n",
       "      <td>hearing_empath</td>\n",
       "      <td>tweet number</td>\n",
       "      <td>retweet number</td>\n",
       "      <td>medieval_empath</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 204 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              0               1            2                3    \\\n",
       "0  morning_empath  wedding_empath  fear_empath  computer_empath   \n",
       "\n",
       "                4              5            6                 7    \\\n",
       "0  furniture_empath  legend_empath  cold_empath  time_diff_median   \n",
       "\n",
       "            8                9    ...           194            195  \\\n",
       "0  sleep_empath  vacation_empath  ...  phone_empath  listen_empath   \n",
       "\n",
       "           196          197                 198             199  \\\n",
       "0  tool_empath  lust_empath  programming_empath  hygiene_empath   \n",
       "\n",
       "              200           201             202              203  \n",
       "0  hearing_empath  tweet number  retweet number  medieval_empath  \n",
       "\n",
       "[1 rows x 204 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "self_feature_importance_rank = np.argsort(node_feature_importance[target_idx])\n",
    "print(np.sum(node_feature_importance[target_idx]))\n",
    "print('The node itself is the {}-th important node'.format(1 + node_importance_rank.tolist().index(target_idx)))\n",
    "df = pd.DataFrame([feature_names[self_feature_importance_rank][::-1]], columns = range(204)) \n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For different nodes, the same features may have different ranks. To understand the overall importance of the features, we now analyze the average feature importance rank for the above selected nodes. Specifically, we obtain the average rank of each specific feature among the top-250 important nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ridicule_empath 17.976\n",
      "furniture_empath 23.304\n",
      "science_empath 29.112\n",
      "sleep_empath 29.892\n",
      "legend_empath 32.992\n",
      "joy_empath 35.012\n",
      "farming_empath 36.728\n",
      "morning_empath 37.02\n",
      "children_empath 40.18\n",
      "hipster_empath 41.76\n",
      "time_diff_median 42.168\n",
      "wedding_empath 43.82\n",
      "anger_empath 43.888\n",
      "time_diff 45.784\n",
      "vacation_empath 48.064\n",
      "traveling_empath 48.528\n",
      "anonymity_empath 51.9\n",
      "work_empath 53.216\n",
      "money_empath 53.936\n",
      "listen_empath 55.0\n",
      "fire_empath 56.668\n",
      "exasperation_empath 57.028\n",
      "cleaning_empath 57.644\n",
      "hiking_empath 58.696\n",
      "ocean_empath 59.456\n",
      "fight_empath 60.072\n",
      "government_empath 61.372\n",
      "programming_empath 61.648\n",
      "divine_empath 62.532\n",
      "hate_empath 68.024\n",
      "computer_empath 70.904\n",
      "weakness_empath 71.74\n",
      "cold_empath 72.312\n",
      "speaking_empath 72.58\n",
      "hygiene_empath 72.964\n",
      "dispute_empath 73.16\n",
      "writing_empath 74.38\n",
      "friends_empath 74.572\n",
      "communication_empath 75.172\n",
      "disappointment_empath 75.768\n",
      "blue_collar_job_empath 75.852\n",
      "politics_empath 76.032\n",
      "office_empath 76.12\n",
      "independence_empath 76.388\n",
      "shame_empath 76.92\n",
      "dance_empath 77.004\n",
      "timidity_empath 77.216\n",
      "school_empath 77.728\n",
      "attractive_empath 77.924\n",
      "positive_emotion_empath 78.16\n",
      "messaging_empath 78.244\n",
      "followers_count 78.928\n",
      "death_empath 78.968\n",
      "number urls 81.18\n",
      "swimming_empath 81.252\n",
      "social_media_empath 83.308\n",
      "college_empath 83.656\n",
      "neglect_empath 84.18\n",
      "favorites_count 84.184\n",
      "childish_empath 84.864\n",
      "crime_empath 84.872\n",
      "kill_empath 85.096\n",
      "violence_empath 85.38\n",
      "zest_empath 85.492\n",
      "sadness_empath 88.004\n",
      "pet_empath 89.624\n",
      "journalism_empath 90.328\n",
      "number hashtags 90.376\n",
      "medical_emergency_empath 90.392\n",
      "rage_empath 90.476\n",
      "philosophy_empath 90.552\n",
      "dominant_personality_empath 90.576\n",
      "healing_empath 91.36\n",
      "royalty_empath 91.624\n",
      "help_empath 91.856\n",
      "negotiate_empath 92.428\n",
      "white_collar_job_empath 92.768\n",
      "monster_empath 93.092\n",
      "movement_empath 93.484\n",
      "pride_empath 95.336\n",
      "poor_empath 95.648\n",
      "car_empath 96.844\n",
      "plant_empath 97.48\n",
      "business_empath 97.616\n",
      "statuses_count 97.656\n",
      "economics_empath 97.692\n",
      "fear_empath 97.8\n",
      "internet_empath 98.172\n",
      "anticipation_empath 98.268\n",
      "shape_and_size_empath 98.704\n",
      "affection_empath 98.972\n",
      "horror_empath 99.156\n",
      "leader_empath 99.86\n",
      "ancient_empath 99.94\n",
      "tool_empath 100.288\n",
      "sympathy_empath 100.46\n",
      "fashion_empath 101.672\n",
      "gain_empath 102.784\n",
      "technology_empath 103.624\n",
      "occupation_empath 104.06\n",
      "banking_empath 104.14\n",
      "emotional_empath 105.388\n",
      "sports_empath 105.688\n",
      "followees_count 106.372\n",
      "urban_empath 106.912\n",
      "worship_empath 107.168\n",
      "lust_empath 108.112\n",
      "politeness_empath 108.248\n",
      "tweet number 108.768\n",
      "appearance_empath 108.776\n",
      "stealing_empath 109.1\n",
      "dominant_heirarchical_empath 109.676\n",
      "vehicle_empath 109.792\n",
      "water_empath 109.852\n",
      "quote number 110.344\n",
      "warmth_empath 110.388\n",
      "body_empath 111.256\n",
      "achievement_empath 111.332\n",
      "wealthy_empath 112.304\n",
      "prison_empath 112.308\n",
      "torment_empath 112.34\n",
      "feminine_empath 112.692\n",
      "driving_empath 113.564\n",
      "retweet number 114.276\n",
      "toy_empath 114.528\n",
      "liquid_empath 114.584\n",
      "youth_empath 114.916\n",
      "art_empath 115.26\n",
      "shopping_empath 115.756\n",
      "listed_count 116.42\n",
      "meeting_empath 117.304\n",
      "law_empath 118.16\n",
      "air_travel_empath 118.192\n",
      "reading_empath 118.744\n",
      "giving_empath 119.212\n",
      "competing_empath 119.348\n",
      "party_empath 119.564\n",
      "health_empath 119.96\n",
      "cooking_empath 120.012\n",
      "animal_empath 121.06\n",
      "real_estate_empath 121.228\n",
      "war_empath 121.244\n",
      "order_empath 121.432\n",
      "family_empath 121.504\n",
      "beauty_empath 121.552\n",
      "sexual_empath 121.692\n",
      "optimism_empath 122.252\n",
      "hearing_empath 122.456\n",
      "fun_empath 122.572\n",
      "medieval_empath 123.016\n",
      "pain_empath 123.732\n",
      "eating_empath 123.996\n",
      "surprise_empath 124.08\n",
      "nervousness_empath 124.236\n",
      "disgust_empath 124.364\n",
      "deception_empath 124.832\n",
      "irritability_empath 124.94\n",
      "cheerfulness_empath 125.264\n",
      "status length 125.672\n",
      "confusion_empath 126.08\n",
      "baddies 126.124\n",
      "tourism_empath 126.432\n",
      "trust_empath 126.528\n",
      "military_empath 127.54\n",
      "phone_empath 127.56\n",
      "religion_empath 128.436\n",
      "exercise_empath 129.712\n",
      "contentment_empath 130.172\n",
      "play_empath 130.244\n",
      "breaking_empath 130.268\n",
      "celebration_empath 130.62\n",
      "mentions 131.332\n",
      "sailing_empath 131.76\n",
      "power_empath 132.156\n",
      "alcohol_empath 132.836\n",
      "smell_empath 133.212\n",
      "swearing_terms_empath 133.32\n",
      "weapon_empath 133.648\n",
      "sound_empath 135.104\n",
      "subjectivity 135.2\n",
      "payment_empath 136.492\n",
      "love_empath 138.336\n",
      "injury_empath 138.664\n",
      "restaurant_empath 139.752\n",
      "strength_empath 140.268\n",
      "night_empath 140.456\n",
      "ship_empath 141.52\n",
      "ugliness_empath 141.84\n",
      "valuable_empath 143.04\n",
      "negative_emotion_empath 143.056\n",
      "clothing_empath 143.98\n",
      "suffering_empath 144.212\n",
      "weather_empath 145.208\n",
      "rural_empath 149.572\n",
      "terrorism_empath 149.784\n",
      "envy_empath 155.2\n",
      "aggression_empath 156.172\n",
      "leisure_empath 160.44\n",
      "noise_empath 161.236\n",
      "beach_empath 161.524\n",
      "masculine_empath 162.468\n",
      "musical_empath 163.676\n",
      "superhero_empath 165.444\n",
      "home_empath 168.604\n"
     ]
    }
   ],
   "source": [
    "from collections import defaultdict\n",
    "average_feature_rank = defaultdict(int)\n",
    "for i in node_importance_rank[:250]:\n",
    "    feature_rank = list(feature_names[np.argsort(node_feature_importance[i])[::-1]])\n",
    "    for j in range(len(feature_rank)):\n",
    "        average_feature_rank[feature_rank[j]] += feature_rank.index(feature_rank[j])\n",
    "for k in average_feature_rank.keys():\n",
    "    average_feature_rank[k] /= 250.0\n",
    "sorted_avg_feature_rank = sorted(average_feature_rank.items(), key=lambda a:a[1])\n",
    "for feat, avg_rank in sorted_avg_feature_rank:\n",
    "    print(feat, avg_rank)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems for our target node, topics relevant to cleaning, hipster, etc. are important while those such as leaisure, ship, goverment, etc. are not important."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then calculate the link importance for the edges that are connected to the target node within k hops (k = 2 for our GCN model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "link_importance = int_saliency.get_integrated_link_masks(target_idx, 0, steps=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 4289572, 2) (1, 4289572)\n"
     ]
    }
   ],
   "source": [
    "(x, y) = link_importance.nonzero()\n",
    "[X,all_targets,A_index, A], y_true_all = all_gen[0]\n",
    "print(A_index.shape, A.shape)\n",
    "G_edge_indices = [(A_index[0, k, 0], A_index[0, k, 1]) for k in range(A_index.shape[1])]\n",
    "link_dict = {(A_index[0, k, 0], A_index[0, k, 1]):k for k in range(A_index.shape[1])}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a sanity check, we expect the most important edge to connect important nodes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There are 26145 edges within the ego graph of the target node\n",
      "The most important edge connects 0-th important node and 13-th important node\n"
     ]
    }
   ],
   "source": [
    "nonzero_importance_val = link_importance[(x,y)].flatten().tolist()[0]\n",
    "link_importance_rank = np.argsort(nonzero_importance_val)[::-1]\n",
    "edge_number_in_ego_graph = link_importance_rank.shape[0]\n",
    "print('There are {} edges within the ego graph of the target node'.format(edge_number_in_ego_graph))\n",
    "x_rank, y_rank = x[link_importance_rank], y[link_importance_rank]\n",
    "print('The most important edge connects {}-th important node and {}-th important node'.format(node_importance_rank.tolist().index(x_rank[0]), (node_importance_rank.tolist().index(y_rank[0]))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To ensure that we are getting the correct importance for edges, we then check what happens if we perturb the top-10 most important edges. Specifically, if we remove the top important edges according to the calculated edge importance scores, we should expect to see the prediction of the target node change. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 4289572, 2)\n",
      "A_perturb.shape = (1, 4289572)\n"
     ]
    }
   ],
   "source": [
    "from copy import deepcopy\n",
    "\n",
    "selected_nodes = np.array([[target_idx]], dtype='int32')\n",
    "prediction_clean = model.predict([X, selected_nodes, A_index, A]).squeeze()\n",
    "A_perturb = deepcopy(A)\n",
    "print('A_perturb.shape = {}'.format(A_perturb.shape))\n",
    "#we remove top 1% important edges in the graph and see how the prediction changes\n",
    "topk = int(edge_number_in_ego_graph * 0.01)\n",
    "\n",
    "for i in range(topk):\n",
    "    edge_x, edge_y = x_rank[i], y_rank[i]\n",
    "    edge_index = link_dict[(edge_x, edge_y)]\n",
    "    A_perturb[0, edge_index] = 0\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, the prediction score drops after the perturbation. The target node is predicted as non-hateful now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The prediction score changes from 0.9631094336509705 to 0.12830840051174164 after the perturbation\n"
     ]
    }
   ],
   "source": [
    "prediction = model.predict([X, selected_nodes, A_index, A_perturb]).squeeze()\n",
    "print('The prediction score changes from {} to {} after the perturbation'.format(prediction_clean, prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTES: For UX team, the above notebook shows how we are able to compute the importance of nodes and edges. However, it seems the ego graph of the target node in twitter dataset is often very big so that we may draw only top important nodes/edges on the visualization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
